{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# xray + dask\n",
    "\n",
    "*This was modified from a notebook originally written by Stephan Hoyer*\n",
    "\n",
    "Weather data -- especially the results of numerical weather simulations -- is big. Some of the biggest super computers make weather forecasts, and they save their output on increasingly high resolution grids. Even for data analysis purposes, it's easy to need to process 10s or 100s of GB of data.\n",
    "\n",
    "There are many excellent tools for working with weather data, which is usually stored in the netCDF file format. Many of these have support for out-of-core data, notably including the command line tools [NCO](http://nco.sourceforge.net/nco.html) and [CDO](https://code.zmaw.de/projects/cdo/wiki/cdo). There are even Python tools, including a netCDF4 library and Iris. However, none of these tools matched the ease of use of pandas. We knew there there was a better way, so we decided to write xray, a library for working with multi-dimensional labeled data.\n",
    "\n",
    "The latest release of xray includes support for processing datasets that don't fit into memory using dask, a new Python library that extends NumPy to out-of-core datasets by blocking arrays into small chunks and using a simple task scheduling abstraction. Dask allows xray to easily process out of core data and simultaneously make use of all our CPUs resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data\n",
    "\n",
    "First, we'll import dask and setup a ThreadPool for processing tasks. Dask currently doesn't do this automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import xray\n",
    "import dask.array as da\n",
    "import numpy as np\n",
    "\n",
    "import dask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the new `xray.open_mfdataset` function to open archived weather data from ECMWF. It opens a glob of netCDF files on my local disk and automatically infers how to combine them into a few logical arrays by reading their metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/mrocklin/data/ecmwf/2014-01-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-01-31.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-02-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-03-31.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-04-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-05-31.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-06-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-07-31.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-08-31.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-09-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-10-31.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-11-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-01.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-02.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-03.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-04.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-05.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-06.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-07.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-08.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-09.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-10.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-11.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-12.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-13.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-14.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-15.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-16.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-17.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-18.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-19.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-20.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-21.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-22.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-23.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-24.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-25.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-26.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-27.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-28.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-29.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-30.nc3\r\n",
      "/home/mrocklin/data/ecmwf/2014-12-31.nc3\r\n",
      "/home/mrocklin/data/ecmwf/test1.nc3\r\n"
     ]
    }
   ],
   "source": [
    "!ls /home/mrocklin/data/ecmwf/*.nc3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "from_array() takes at least 2 arguments (2 given)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-2-c6549176712c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mds\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mxray\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mopen_mfdataset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'/home/mrocklin/data/ecmwf/*.nc3'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'scipy'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/xray-0.4.1_55_g651d48d-py2.7.egg/xray/backends/api.pyc\u001b[0m in \u001b[0;36mopen_mfdataset\u001b[1;34m(paths, blockshape, concat_dim, **kwargs)\u001b[0m\n\u001b[0;32m    182\u001b[0m     \u001b[0mdatasets\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mopen_dataset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mp\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mp\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mpaths\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    183\u001b[0m     \u001b[0mfile_objs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mds\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_file_obj\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mds\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdatasets\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 184\u001b[1;33m     \u001b[0mdatasets\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mds\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreblock\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mblockshape\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mblockshape\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mds\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdatasets\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    185\u001b[0m     \u001b[0mcombined\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mauto_combine\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdatasets\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mconcat_dim\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mconcat_dim\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    186\u001b[0m     \u001b[0mcombined\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_file_obj\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_MultiFileCloser\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfile_objs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/xray-0.4.1_55_g651d48d-py2.7.egg/xray/core/dataset.pyc\u001b[0m in \u001b[0;36mreblock\u001b[1;34m(self, blockdims, blockshape)\u001b[0m\n\u001b[0;32m    923\u001b[0m                                                 name=k)\n\u001b[0;32m    924\u001b[0m                                       if v.ndim > 0 else v))\n\u001b[1;32m--> 925\u001b[1;33m                                  for k, v in self.variables.items()])\n\u001b[0m\u001b[0;32m    926\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_replace_vars_and_dims\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvariables\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    927\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/xray-0.4.1_55_g651d48d-py2.7.egg/xray/core/variable.pyc\u001b[0m in \u001b[0;36mreblock\u001b[1;34m(self, blockdims, blockshape, name)\u001b[0m\n\u001b[0;32m    466\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    467\u001b[0m             data = da.from_array(data, blockdims=blockdims,\n\u001b[1;32m--> 468\u001b[1;33m                                  blockshape=blockshape, name=name)\n\u001b[0m\u001b[0;32m    469\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    470\u001b[0m         return type(self)(self.dims, data, self._attrs, self._encoding,\n",
      "\u001b[1;31mTypeError\u001b[0m: from_array() takes at least 2 arguments (2 given)"
     ]
    }
   ],
   "source": [
    "ds = xray.open_mfdataset('/home/mrocklin/data/ecmwf/*.nc3', engine='scipy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11 GB of Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11.324758529663086"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.prod(ds.dims.values()) * 8 * 2 ** -30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4GB of Memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MemTotal:        3766528 kB\r\n"
     ]
    }
   ],
   "source": [
    "!cat /proc/meminfo | grep MemTotal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Index with meaningful values, not numbers "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xray.Dataset>\n",
       "Dimensions:   (latitude: 721, time: 1464)\n",
       "Coordinates:\n",
       "  * latitude  (latitude) >f4 90.0 89.75 89.5 89.25 89.0 88.75 88.5 88.25 88.0 87.75 87.5 87.25 ...\n",
       "  * time      (time) datetime64[ns] 2014-01-01 2014-01-01T06:00:00 2014-01-01T12:00:00 ...\n",
       "Data variables:\n",
       "    u10       (time, latitude) float64 0.8483 0.9731 1.098 1.037 1.258 1.459 1.39 1.322 1.203 ...\n",
       "    v10       (time, latitude) float64 -0.282 -0.2981 -0.3142 -0.3064 -0.3349 -0.3596 -0.3326 ...\n",
       "    t2m       (time, latitude) float64 254.0 253.9 253.8 253.9 253.8 253.6 253.4 253.1 252.8 ..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# x.mean(2)\n",
    "ds.mean('longitude')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xray.Dataset>\n",
       "Dimensions:    (latitude: 360, longitude: 1440, time: 120)\n",
       "Coordinates:\n",
       "  * longitude  (longitude) >f4 0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 2.75 3.0 3.25 ...\n",
       "  * latitude   (latitude) >f4 90.0 89.75 89.5 89.25 89.0 88.75 88.5 88.25 88.0 87.75 87.5 87.25 ...\n",
       "  * time       (time) datetime64[ns] 2014-04-01 2014-04-01T06:00:00 2014-04-01T12:00:00 ...\n",
       "Data variables:\n",
       "    u10        (time, latitude, longitude) float64 2.715 2.683 2.65 2.616 2.583 2.551 2.518 ...\n",
       "    v10        (time, latitude, longitude) float64 -7.506 -7.511 -7.517 -7.523 -7.529 -7.535 ...\n",
       "    t2m        (time, latitude, longitude) float64 254.1 254.1 254.1 254.1 254.1 254.1 254.1 ...\n",
       "Attributes:\n",
       "    Conventions: CF-1.0\n",
       "    history: 2015-02-11 16:45:42 GMT by grib_to_netcdf-1.13.0: grib_to_netcdf /data/data01/mars-web230-20150211164538-19989-9003.target -o /data/data01/mars-web230-20150211164541-19989-9004.nc"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds.sel(time=\"2014-04\", latitude=(ds.latitude > 10 & (ds.latitude < 40)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groupby operations and datetime handling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5min 3s, sys: 36.4 s, total: 5min 39s\n",
      "Wall time: 1min 53s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<xray.Dataset>\n",
       "Dimensions:    (latitude: 721, longitude: 1440, month: 12)\n",
       "Coordinates:\n",
       "  * longitude  (longitude) >f4 0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 2.75 3.0 3.25 ...\n",
       "  * latitude   (latitude) >f4 90.0 89.75 89.5 89.25 89.0 88.75 88.5 88.25 88.0 87.75 87.5 87.25 ...\n",
       "  * month      (month) int64 1 2 3 4 5 6 7 8 9 10 11 12\n",
       "Data variables:\n",
       "    u10        (month, latitude, longitude) float64 -2.758 -2.749 -2.74 -2.731 -2.722 -2.713 ...\n",
       "    v10        (month, latitude, longitude) float64 1.707 1.718 1.729 1.741 1.752 1.764 1.775 ...\n",
       "    t2m        (month, latitude, longitude) float64 252.4 252.4 252.4 252.4 252.4 252.4 252.4 ..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time ds.groupby('time.month').mean('time').load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bandwidth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "97.34513274336283"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "11e9 / 113 / 1e6 # MB/s"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
